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1. Introduction 


Collecting and analysing students’ opinions on their learning experiences during enrollment 
in an academic program is widely recognised as a key strategy for evaluating tertiary educa- 
tion quality. Academic institutions require students to participate every year in specific surveys, 
aiming to gather their viewpoints about the organisation of the single courses and their feel- 
ings about the teaching activity’s traits and effectiveness. The Standards and guidelines for 
quality assurance in the European Higher Education Area (ESG, 2015), for example, under- 
line the relevance of students’ voices in the assessment processes. Likewise, students’ and 
graduates’ opinions constitute essential information for the quality assurance of the Italian uni- 
versity system. The National Agency for the Evaluation of Universities and Research Institutes 
(ANVUR), to harmonise the data collection in all the universities, provides guidelines to build 
the questionnaire administered in the surveys on students’ opinions. Initially released in 2013, 
these guidelines were updated in 2017 and are currently in use. The survey, under article 1 
of Law 370/1999, is mandatory and autonomously carried out every academic year by the dif- 
ferent institutions, representing one of the fundamental sources for the so-called AVA system! 
(i.e., self-assessment, periodic assessment, accreditation), introduced in Italy by Law 240/2010 
and Legislative Decree 19/2012. This system aims to improve teaching and research quality in 
universities, applying a model based on internal planning and management procedures. 

This study investigates students’ satisfaction towards courses delivered at the University of 
Calabria, focusing on the academic year 2020-2021. In this period, almost corresponding to 
the occurrence of the second and third waves of COVID-19 outbreak in Italy, traditional teach- 
ing methods were substantially disrupted by the social distancing actions pursued by the Italian 
government, enhancing the use of blended and hybrid learning (e.g., Aboagye et al., 2021; 
Chaturvedi et al., 2021). Here we considered the first-year courses since students enrolled in 
2020-2021 programs experienced the completion of their previous educational degrees in the 
first wave of COVID-19. We carried out a latent class analytical strategy to profile students’ 
satisfaction at a course level, taking into account their interest in each course and their per- 
ceptions about the course organisation and the instructor’s behaviour. Since the items listed 
in the survey are expressed as 4-point balanced scales, we used the so-called Latent Profile 
Analysis (LPA) to identify unobserved course profiles, starting from students’ responses to the 
continuous indicators concerning course satisfaction. 


2. Theoretical framework and data structure 


LPA is a statistical approach belonging to finite mixture models. It can be seen as a variant 
of latent class analysis (LCA, Oberski, 2016) aiming at identifying a set of discrete, exhaustive, 
and non-overlapping groups of subjects characterised by different patterns of responses on in- 
dicator items, typically represented by ordinal or continuous manifest variables. Each subject 
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is assigned to the most likely latent group, i.e. an unobserved profile that generates patterns 
of responses on the indicators. LPA may be considered, therefore, a case-centred analytic tool 
focusing on similarities and differences among subjects rather than relations among variables 
(Bergman and Magnusson, 1997). 

Assuming that the continuous (or ordinal) variables are normally distributed within each 
latent profile, a model of G components aims at representing the distributions of the observed 
subjects’ scores on a set of indicator items x; (i = 1,..., n), given the latent categorical variable 
O, as a function of the probability of the subjects to be typed into a profile and the profile- 
specific normal density: 


G 
flO) = SO te fe (xix) (1) 
k=1 


where m, and @;, represent the probability of belonging to the k-th latent profile (with my sum- 
ming to | across the different profiles) and the estimation of the mean and the set of vari- 
ances/covariances for k, respectively (Tein et al., 2013). 

LPA has been recently used in the educational domain, for example, to identify students’ 
time use profiles (Fosnacht et al., 2018), to explore motivation patterns in learning environments 
(Hodis and Hodis, 2020), to define types of social support for student resilience during the 
COVID-19 pandemic (Mai et al., 2021). In the following, LPA is used to identify academic 
course profiles, considering students’ opinions about the courses they attended. 

The yearly survey about students’ satisfaction carried out at the University of Calabria (Italy) 
was used to build a dataset of course response patterns. The questionnaire administered in the 
survey is built following the ANVUR guidelines to harmonise the data collection in all the 
different Universities. 

We focused, in particular, on the academic year 2020-2021. During this year, because of 
the COVID-19 pandemic, courses have been delivered in presence, in distance (via online plat- 
forms), or by mixing the two types. The total number of the collected questionnaires was 77,049 
for the courses of 73 academic programs included in the entire university catalogue. After fil- 
tering only the questionnaires completed by first-year students that attended at least half of the 
lectures for each first-year course (24,064), we selected 10 different items concerning for each 
course the Interest of the students, the Instructor behaviour and the Teaching characteristics. 
Table 1 lists the three dimensions and the corresponding items. 


Table 1: Dimensions and indicator items. 


Dimension Item Short label 
Interest towards the course INT_1 
Interest (INTE) Interest stimulated by the instructor INT_2 
Instructor clear in the explanations INS_1 
Instructor INST) Instructor available for tutoring INS_2 
Instructor on time at the lectures INS 3 
Course coherent with the syllabus TEA_1 
Examination rules clearly defined TEA-2 
Teaching (TEAC) Adequate prior knowledge for the course TEA3 
Adequate learning material TEA_4 
Workload proportional to ECTS TEA5 


Since the items listed in the survey are expressed as 4-point balanced scales (definitively no, 
more no than yes, more yes than no, definitively yes), we converted the ranks in scores from 1 to 
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4. Moreover, since the items INT_1 and INS_3 showed a certain number of missing data (10.8% 
and 6.7%, respectively), we performed a multiple imputation procedure in order to save all the 
selected cases in the dataset. Finally, the cases were collapsed at a course level by averaging 
the individual response patterns, and hence the course response patterns were standardised. 
The resulting 657 x 10 matrix was used in the analysis. The imputation of missing data was 
performed by using the R library mice, whereas LPA was performed by using the library mclust 
(van Buuren and Groothuis-Oudshoorn, 2011; Scrucca et al., 2016). 


3. Model selection and main findings 


A key question in finite mixture modelling is how many latent classes should be included. 
The selection of the best model was carried out by jointly evaluating the Bayesian Informa- 
tion Criterion (BIC) and the Integrated Complete-data Likelihood (ICL) criterion proposed by 
Biernacki et al. (2000). ICL appears more robust than BIC, adding a penalty on solutions with 
greater entropy or classification uncertainty. 

In addition to the number of profiles, the model can be specified in terms of whether and how 
the variable variances and covariances are estimated. Geometric features (shape, volume, ori- 
entation) of the clusters are determined by the covariances. We estimated two kinds of models, 
considering in both cases profiles with equal volume and shape. In the EEI model, the indi- 
cator variables are set to have zero covariances within and across profiles. Indicator-variable 
variances are allowed to vary within profiles but are constrained to be equal between them. 
In the EEE model, the complete variable (co)variance matrix is estimated, with variances and 
covariances constrained to be the same across the profiles. 

Table 2 shows the fit indices of the two models, including the log-likelihood £ (with the 
corresponding degrees of freedom), the BIC and the ICL. 


Table 2: Data fit of EEI and EEE models. 


EEI EEE 
N. of profiles £ df BIC ILC | l df BIC ILC 
1 -9317.31 20 -18764.37 -14304.68 | -6941.49 65 -18764.37 -14304.68 


-8081.98 31 -16365.07 -16404.11 | -6871.95 76 -14236.97 -14293.81 
-7553.06 42 -15378.61 -15444.33 | -6765.99 87 -14096.41 -14160.06 
-7304.36 53 -14952.57 -15051.02 | -6628.85 98 -13893.49 -13912.09 
-7188.40 64 -14792.00 -14927.39 | -6640.24 109 -13987.64 -14472.57 


AUN 


The log-likelihood had lower values for each model that increased of one latent profile, with 
EEE models showing lower values than EEI models. Regarding the information criteria, we 
observed that the two indices confirmed that EEE models are better than EEI models. Jointly 
considering the results of the three selection criteria listed above, we selected the EEE model 
with 4 latent clusters. To validate our choice, we performed a bootstrap likelihood ratio test 
(BLRT) to verify the null hypothesis that a (k + 1)-profile model is equal to or better than a 
k-profile model, i.e. that an increase in the number of profiles increases fit (Nylund et al., 2007). 
Table 3 shows the results of the test and the corresponding p-values, suggesting that a 4-profile 
solution is optimal. 

The four profiles allowed to identify different satisfaction patterns for the first-years courses 
under investigation. Table 4 reports the mean values of the different items per profile, looking 
at each as a factor score with a mean equal to O (due to standardisation). 

Profile 1 (90.26% of the courses) showed for each item a score above 0, identifying courses 
with an average level of student satisfaction. Profile 2 (5.48% of courses) showed for the Interest 
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Table 3: BLRT for the EEE models. 
| lvs2 2 vs 3 3vs4 4vs5 


LRTs | 139.072 211.926 274.282 -22.787 
p-value | 0.001* 0.001* 0.001* 0.217 


Table 4: Mean values of the ten items per profile. 


Item Profile 1 Profile2 Profile3 Profile 4 

INTE Interest towards the course 0.01 0.14 -0.08 -0.68 
Interest stimulated by the instructor 0.05 0.16 -1.21 -1.18 
Instructor clear in the explanations 0.07 -0.07 -1.10 -1.53 

INST Instructor available for tutoring 0.07 0.14 -0.58 -3.77 
Instructor on time at the lectures 0.12 0.14 -3.60 -1.17 

Course coherent with the syllabus 0.04 0.12 -0.94 -1.05 
Examination rules clearly defined 0.03 0.03 -0.77 -0.61 

TEAC Adequate prior knowledge for the course 0.02 -0.16 0.37 -0.98 
Adequate learning material 0.06 -0.56 -0.59 -0.60 
Workload proportional to ECTS 0.14 -2.13 0.17 -0.38 

% membership 90.26 5.48 2.74 1.52 


and Instructor dimensions — as well as the items of the Teaching dimension related to course or- 
ganisation — scores above 0, with values greater than the corresponding values of Profile 1. The 
only negative value was observed for the clarity of instructors, with a value just below 0. Nev- 
ertheless, the courses likely belonging to this profile showed negative scores for the adequacy 
of the prior knowledge required to attend the lectures and the learning materials, with a peak 
for the course workload that is not perceived as proportional to the ECTS. Profile 3 (2.74% of 
the courses) is characterised by a fair share of students’ dissatisfaction, particularly concerning 
the instructors’ activities. Courses likely belonging to this profile showed very negative scores 
for the interest stimulated by the instructor, the clarity of this latter in the explanations, and a 
peak for the punctuality of the instructors. On the other hand, the profile had positive scores for 
the adequacy of the prior knowledge and the workload. Profile 4 (1.52% of the courses), finally, 
was even more characterised by dissatisfaction, with scores significantly above 0 for all the 
items. In particular, the Instructor dimension showed very negative scores, together with a low 
score for the interest stimulated by the instructors themselves. At the same time, we observed 
negative scores for the items of the Teaching dimension, with an unfavourable evaluation of the 
coherence of the courses with respect to their syllabus and an inadequacy of prior knowledge 
and learning materials. A joint lecture on the profile membership and some covariates can offer 
valuable insights into the less satisfying courses. 

To characterise the profiles, we focused at this stage of the research only on the nature 
of academic programs in which courses are embodied, taking into account if the courses be- 
long to undergraduate and single-cycle programs (namely, Laurea and Laurea Magistrale a 
Ciclo Unico) or to master programs (namely, Laurea Magistrale). Focusing on the profiles that 
showed a certain degree of dissatisfaction, we observed that 83.3% of courses in Profile 2 are 
embodied in master programs. At the same time, the 66.7% and the 50.0% of courses in Profile 
3 and Profile 4, respectively, are embodied in master programs. This aspect can help charac- 
terise the source of the satisfaction (or dissatisfaction) perceived by students, helping academic 
institutions perform targeted interventions on the courses showing specific shortcomings. Con- 
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sidering, for example, the courses belonging to Profile 2, preparing more effective learning 
materials and re-designing the course programs may improve students’ satisfaction, taking into 
account the higher expectations of master students. Nevertheless, the data used in this study 
referred to the academic year 2020-2021, which occurred during the second and third waves of 
the COVID-19 pandemic in Italy. This means that a comparison with other academic years is 
necessary to detect potential structural weaknesses that deserve greater attention. 


4. Final remarks and future research 


This study analysed students’ satisfaction towards courses attended during the first year of 
the academic programs delivered at the University of Calabria. The different course types were 
depicted using a latent class approach known as LPA, a finite mixture model able to measure the 
impact of an unobserved categorical variable defining different latent profiles on a set of contin- 
uous variables. The survey structure did not allow us to evaluate satisfaction at a student-level 
since each questionnaire is registered with a different ID due to the privacy policies imple- 
mented at the University of Calabria. For this reason, we evaluated satisfaction at a course 
level, trying to identify different course types and analyse their characteristics. By contrast, the 
data averaging did not allow considering the variability of students’ opinions for each course, 
posing a possible limitation of the present study. 

The response patterns expressing students’ satisfaction/dissatisfaction levels to the different 
aspects concerning teaching and the organisation of learning activities characterised a 4-profile 
model, revealing for each one the most critical aspects. Remarkably, together with a profile 
encompassing the majority of courses and revealing a general degree of satisfaction, we identi- 
fied three course profiles expressing a different degree of dissatisfaction instead. A focus on the 
academic programs the latter courses belong — considering if they were included in first-cycle, 
single-cycle or second-cycle programs — showed that graduate students have potentially higher 
expectations than undergraduate students, evaluating in a more critical way the course organ- 
isation and the workload required by each course. A noteworthy aspect is that these students 
experienced the rapid change of teaching induced by COVID-19 during the last year of their 
first-cycle degree, starting a new cycle of study in the uncertainty caused by the ongoing social 
distancing and the limitations established by the Italian government. 

The analytical strategy employed in the study can be easily implemented as a visual tool 
helping academic institutions at a department level (e.g., by the Joint Teaching-Student Com- 
mittees) or at a university level (e.g., by the Independent Evaluation Units) in the quality assur- 
ance systems, giving a hint of which courses have to be carefully monitored and at the same time 
of which aspect are perceived as more critical by the students. Currently, a different version of 
the survey has been tested in some universities by the National Agency for the Evaluation, and 
it is ready to be released in the short term a new version of the guidelines. 

Several future developments of the study can be considered. First, the effect of some meta- 
data (as covariates) may help better define and characterise the profiles, taking into account the 
cycle dimension and the attendance rate, and the domain of the courses (e.g., courses based on 
quantitative or qualitative methods). Second, a longitudinal analysis may help in evaluating how 
a shock like the COVID-19 pandemic or a change in the academic programs’ regulation influ- 
ences the perception of courses’ quality, estimating the transition of a course (in a probabilistic 
fashion) from a latent profile to another one in different periods (Collins and Lanza, 2013). This 
latter variant of latent models can enrich the analytical strategy’s informative power, allowing 
for evaluating the quality across time. 
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